Repository navigation
feat(cluster): move etcd's peer traffic onto the servers' LAN addresses - #257
Merged
Merged
Conversation
The probe timed a TCP connect to every voter's etcd peer port at the node's InternalIP, which is where etcd listens only while its member uses the mesh address. It now dials the address k3s keeps in the etcd.k3s.cattle.io/node-address annotation, the one the member's peer URL is built from, so the reading stays right for a member that has moved onto its LAN address.
A server whose mesh device stops working loses its etcd peers with it, even when every server shares a LAN, because etcd's peer and client URLs are mesh addresses. github_runner_cluster_etcd_lan_address (per server) and github_runner_cluster_etcd_peers_over_lan (per group) put them on the LAN instead, through k3s's --node-ip, which with --vpn-auth moves only etcd: the node IP, flannel and the API stay on the mesh. No --etcd-arg is used, so k3s's temporary etcd is untouched. k3s registers one peer URL per member and etcd refuses a join whose member URLs differ, so a member has one address and the fallback to the mesh is decided per run: a member stays on the mesh while another server does not reach its LAN address or lacks its own LAN address in its peer certificate. Setting the address only adds it to the certificates. With the switch on, each run moves at most one member (etcdctl member update, then a recreate onto the new address) after checking every server Ready, every member healthy and no read-index timeouts in the servers' logs, and checks the same afterwards; a move that fails is updated back, .env restored and the server restarted on its old address. Turning the switch off moves members back one per run. The address must be on a Docker node on the host network: on the bridge network the VM's NAT rewrites the peer's source address, which fails etcd's certificate check, so the role refuses it, and refuses native nodes and headscale_in_cluster. A host that sets neither keeps the same k3s arguments, .env and Compose files as before.
… mesh The etcd_lan scenario runs three servers on the host network of Docker daemons sharing the test network. It first shows that cutting one server's Tailscale interface takes its member out of etcd while etcd is on the mesh, then sets the LAN addresses (no member moves), turns github_runner_cluster_etcd_peers_over_lan on and expects one member on the LAN per run with writes succeeding throughout and a rerun restarting nothing, repeats the cut and expects every member healthy and the cut server writable, and turns the switch off to move them back one per run.
etcdctl endpoint health exits non-zero whenever a member is unhealthy, so under pipefail the scenario's check took the case it looks for as not yet. Its output is now read whatever its status.
…sses The check refused real addresses in the mesh test. It now parses the Docker host's addresses as the host network's own LAN check does and names the addresses it found when it refuses.
grep -q stops reading at its first match, so the echo feeding it fails with a broken pipe and pipefail fails the check that the report was made.
Each run moves one server's member and recreates its container, which takes its own API server down; writes through it measured that, not etcd's quorum. Each run now writes through the next server instead.
The etcd_lan scenario starts a writer per move, and each began at the same names, so the second one's writes failed as already existing.
|
🎉 This PR is included in version 3.3.0 🎉 The release is available on:
Your semantic-release bot 📦🚀 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs #256
When a server's mesh device stops working (an expired node key, for example), its etcd member loses its peers even if every server is on the same LAN, because etcd's peer and client URLs are mesh addresses. This adds an opt-in path that puts them on the servers' LAN addresses.
github_runner_cluster_etcd_lan_address(per server, explicit rather than detected, since a host can have several LAN addresses and the one that matters is the one the others reach) andgithub_runner_cluster_etcd_peers_over_lan(per group). The move goes through k3s's--node-ip, which with--vpn-authchanges only etcd's address: the node IP, flannel and the API stay on the mesh. No--etcd-argis used, so the temporary etcd k3s starts on every restart is untouched (the known issue indocs/non-obvious-behaviour.md).k3s registers one peer URL per member and etcd refuses a join whose member URLs do not match, so a member cannot carry both addresses. The fallback to the mesh is therefore decided per run: a member stays on the mesh while another server cannot reach its LAN address, or another server's peer certificate does not yet name its own LAN address.
Limitation: it needs each server to be a Docker node on the host network of a Docker host that has the LAN address itself (a bridged VM on macOS). On the bridge network the VM's NAT rewrites the peer's source address, which fails etcd's peer certificate check, so the role refuses that, and also refuses native nodes and
headscale_in_cluster. All the checks run before anything changes,--checkincluded.Rollout, all in the role README's "etcd peers over the LAN" section:
github_runner_cluster_etcd_peers_over_lan: trueand run once per server. Each run moves at most one member. Before moving it checks that every server is Ready, every member is healthy, there have been no read-index timeouts in any server's log for the last 10 minutes, every other server reaches the address and every other certificate names its own address. Then it runsetcdctl member update, recreates the server on the new address, and checks Ready, membership, health and a 60 s quiet log window afterwards. If anything fails, the member is updated back,.envrestored and the server restarted on its old address, and the run fails..envstill starts etcd on it is refused.A host that sets neither gets the same k3s arguments,
.envand Compose files as before (tested exactly). The role controller's probe now dials theetcd.k3s.cattle.io/node-addressannotation instead of the InternalIP, so its peer reading stays right after a move.Nothing was applied to any live cluster. To opt in, the operator sets the addresses in the fleet's host_vars and follows the steps above, one run per stage.
Tested: the new unit modules (
test_cluster_etcd_lan.pyandtest_cluster_etcd_lan_role.py, the second with real certificates read by openssl), the full unit suite in CI, plus shellcheck, yamllint, the playbook syntax check and ansible-lint (production). I made 15 deliberate breakages to the entrypoint,env.j2, the Compose file list, the filters, the decision tasks and the probe, and the tests caught every one. The newetcd_lanmesh scenario passes in CI. It runs three servers on the host network of dind daemons that share one network, cuts one server's Tailscale interface and sees its member drop out of etcd, then moves every member onto the LAN one run at a time while writing through a server that is not moving. A rerun restarts nothing. The same cut then leaves every member healthy and the cut server writable, and the scenario finishes by moving the members back.What no test here covers is a real Colima bridged VM on macOS and a real expired node key.
role_swapfailed once on this branch with "voter grtest-native is connected to 3 of its 2 peers", the same refusal seen on another branch before this change. It comes from etcd's metrics, not the address the probe dials, and is tracked in #259. I also filed #258 while reading k3s's source: the docs' 100/1000 ms default etcd timing is 500/5000 ms in v1.34.1.